Skip to content

[AMORO-3853] Support to build amoro with Spark-3.5 + Java17 #3859

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 6 commits into
base: master
Choose a base branch
from
Draft

[AMORO-3853] Support to build amoro with Spark-3.5 + Java17 #3859

wants to merge 6 commits into from

Conversation

@turboFei
Copy link
Member

@turboFei turboFei commented Oct 31, 2025 •
edited
Loading

Why are the changes needed?

Close #3853

Brief change log

  • Support to build amoro with Spark-3.5 + Java17

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@github-actions github-actions bot added module:mixed-flink Flink moduel for Mixed Format type:infra type:build labels Oct 31, 2025
@turboFei turboFei changed the title Spark 35 jdk17 [AMORO-3853] Support to build amoro with Spark-3.5 + Java17 Oct 31, 2025
@turboFei turboFei marked this pull request as draft October 31, 2025 19:25
@turboFei
Copy link
Member Author

turboFei commented Oct 31, 2025

Error:

2025-10-31T21:27:36.7841941Z [ERROR] TestKeyedContinuousOptimizing.run  Time elapsed: 0.65 s  <<< ERROR!
2025-10-31T21:27:36.7842810Z java.lang.IllegalStateException: Not an instance of java.lang.Long: 2019-01-26T21:27:34.557898
2025-10-31T21:27:36.7843640Z 	at org.apache.iceberg.data.GenericRecord.get(GenericRecord.java:123)
2025-10-31T21:27:36.7844403Z 	at org.apache.amoro.io.writer.RecordWithAction.get(RecordWithAction.java:68)
2025-10-31T21:27:36.7845027Z 	at org.apache.iceberg.types.Comparators$StructLikeComparator.compare(Comparators.java:111)
2025-10-31T21:27:36.7845597Z 	at org.apache.iceberg.types.Comparators$StructLikeComparator.compare(Comparators.java:84)
2025-10-31T21:27:36.7846143Z 	at org.apache.iceberg.util.StructLikeWrapper.equals(StructLikeWrapper.java:90)
2025-10-31T21:27:36.7846570Z 	at java.base/java.util.HashMap.getNode(HashMap.java:570)
2025-10-31T21:27:36.7846936Z 	at java.base/java.util.HashMap.containsKey(HashMap.java:594)
2025-10-31T21:27:36.7847297Z 	at java.base/java.util.HashSet.contains(HashSet.java:205)
2025-10-31T21:27:36.7847701Z 	at org.apache.iceberg.util.StructLikeSet.contains(StructLikeSet.java:61)
2025-10-31T21:27:36.7848281Z 	at org.apache.amoro.server.optimizing.flow.view.KeyedTableDataView.match(KeyedTableDataView.java:234)
2025-10-31T21:27:36.7849179Z 	at org.apache.amoro.server.optimizing.flow.checker.DataConcurrencyChecker.check(DataConcurrencyChecker.java:70)
2025-10-31T21:27:36.7849932Z 	at org.apache.amoro.server.optimizing.flow.CompleteOptimizingFlow.check(CompleteOptimizingFlow.java:159)
2025-10-31T21:27:36.7850657Z 	at org.apache.amoro.server.optimizing.flow.CompleteOptimizingFlow.optimize(CompleteOptimizingFlow.java:126)
2025-10-31T21:27:36.7851685Z 	at org.apache.amoro.server.optimizing.flow.TestKeyedContinuousOptimizing.mustFullCycle(TestKeyedContinuousOptimizing.java:172)
2025-10-31T21:27:36.7852569Z 	at org.apache.amoro.server.optimizing.flow.TestKeyedContinuousOptimizing.run(TestKeyedContinuousOptimizing.java:136)
2025-10-31T21:27:36.7853244Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2025-10-31T21:27:36.7853848Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
2025-10-31T21:27:36.7854547Z 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2025-10-31T21:27:36.7855198Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
2025-10-31T21:27:36.7855672Z 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
2025-10-31T21:27:36.7856231Z 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2025-10-31T21:27:36.7856794Z 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
2025-10-31T21:27:36.7857346Z 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2025-10-31T21:27:36.7857889Z 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2025-10-31T21:27:36.7858411Z 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2025-10-31T21:27:36.7858934Z 	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
2025-10-31T21:27:36.7859379Z 	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
2025-10-31T21:27:36.7859862Z 	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
2025-10-31T21:27:36.7860347Z 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
2025-10-31T21:27:36.7860833Z 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
2025-10-31T21:27:36.7861561Z 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
2025-10-31T21:27:36.7862041Z 	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
2025-10-31T21:27:36.7862434Z 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
2025-10-31T21:27:36.7862858Z 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
2025-10-31T21:27:36.7863361Z 	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
2025-10-31T21:27:36.7863772Z 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
2025-10-31T21:27:36.7864173Z 	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
2025-10-31T21:27:36.7864526Z 	at org.junit.runners.Suite.runChild(Suite.java:128)
2025-10-31T21:27:36.7864846Z 	at org.junit.runners.Suite.runChild(Suite.java:27)
2025-10-31T21:27:36.7865191Z 	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
2025-10-31T21:27:36.7865581Z 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
2025-10-31T21:27:36.7865991Z 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
2025-10-31T21:27:36.7866411Z 	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
2025-10-31T21:27:36.7866815Z 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
2025-10-31T21:27:36.7867238Z 	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
2025-10-31T21:27:36.7867686Z 	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
2025-10-31T21:27:36.7868073Z 	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
2025-10-31T21:27:36.7868576Z 	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
2025-10-31T21:27:36.7868974Z 	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
2025-10-31T21:27:36.7869335Z 	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
2025-10-31T21:27:36.7869662Z 	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
2025-10-31T21:27:36.7870094Z 	at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
2025-10-31T21:27:36.7870681Z 	at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
2025-10-31T21:27:36.7871248Z 	at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
2025-10-31T21:27:36.7872048Z 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
2025-10-31T21:27:36.7872807Z 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
2025-10-31T21:27:36.7873557Z 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
2025-10-31T21:27:36.7874420Z 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
2025-10-31T21:27:36.7875270Z 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
2025-10-31T21:27:36.7876088Z 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
2025-10-31T21:27:36.7876740Z 	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
2025-10-31T21:27:36.7877281Z 	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
2025-10-31T21:27:36.7877957Z 	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
2025-10-31T21:27:36.7878630Z 	at org.apache.maven.surefire.junitplatform.LazyLauncher.execute(LazyLauncher.java:55)
2025-10-31T21:27:36.7879277Z 	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:223)
2025-10-31T21:27:36.7880043Z 	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:175)
2025-10-31T21:27:36.7880792Z 	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:139)
2025-10-31T21:27:36.7881745Z 	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:456)
2025-10-31T21:27:36.7882712Z 	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:169)
2025-10-31T21:27:36.7883556Z 	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:595)
2025-10-31T21:27:36.7884482Z 	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:581)

Related iceberg issue:

apache/iceberg#6510

@codecov-commenter
Copy link

codecov-commenter commented Oct 31, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 22.13%. Comparing base (4382d55) to head (9ba1f9b).
⚠️ Report is 7 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (4382d55) and HEAD (9ba1f9b). Click for more details.

HEAD has 3 uploads less than BASE
Flag BASE (4382d55) HEAD (9ba1f9b)
core 3 0
Additional details and impacted files 
@@             Coverage Diff              @@
##             master    #3859      +/-   ##
============================================
- Coverage     29.00%   22.13%   -6.87%     
+ Complexity     3850     2461    -1389     
============================================
  Files           631      444     -187     
  Lines         50770    40883    -9887     
  Branches       6543     5766     -777     
============================================
- Hits          14724     9050    -5674     
+ Misses        34989    31075    -3914     
+ Partials       1057      758     -299
Flag Coverage Δ
core ?
trino 22.13% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions bot added the module:ams-server Ams server module label Oct 31, 2025
@turboFei turboFei force-pushed the spark_35_jdk17 branch 2 times, most recently from 1f443ad to 8f8a8e9 Compare October 31, 2025 23:51
@github-actions github-actions bot added module:ams-server Ams server module and removed module:ams-server Ams server module labels Oct 31, 2025
@turboFei turboFei force-pushed the spark_35_jdk17 branch 3 times, most recently from 5f5ec8c to 03e4cb0 Compare November 1, 2025 04:03
@github-actions github-actions bot removed the module:ams-server Ams server module label Nov 1, 2025
@turboFei turboFei closed this Nov 1, 2025
@turboFei turboFei reopened this Nov 1, 2025
@github-actions github-actions bot added the module:ams-server Ams server module label Nov 1, 2025
@turboFei turboFei closed this Nov 1, 2025
@turboFei turboFei reopened this Nov 2, 2025
@turboFei turboFei force-pushed the spark_35_jdk17 branch 4 times, most recently from 7fab1bf to 33d9cae Compare November 2, 2025 23:41
@turboFei
java17
0bc4fa5 
scala

test

apache 34

revert apache 34

java8 compatiblity

compatible jdk11

java version 11

opens

<iceberg.version>1.10.0</iceberg.version>

revert iceberg 1.10.0

debug

Revert "debug"

This reverts commit 03e4cb0.

test

test

ignore
@turboFei
Copy link
Member Author

turboFei commented Nov 3, 2025

#3848 (comment)

amoro_spark3-5.log any comments? @zhoujinsong @xxubai

I also tried compiling it with JDK 17 before. From what I remember, the issue might be caused by inconsistencies in how different JDK versions serialize certain Java classes.

@turboFei turboFei closed this Nov 3, 2025
@turboFei turboFei deleted the spark_35_jdk17 branch November 3, 2025 08:37
@turboFei turboFei restored the spark_35_jdk17 branch November 3, 2025 08:37
@turboFei turboFei reopened this Nov 3, 2025
@github-actions github-actions bot removed the module:ams-server Ams server module label Nov 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

module:mixed-flink Flink moduel for Mixed Format type:build type:infra

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Improvement]: Support to build amoro on java 17

3 participants

@turboFei @codecov-commenter @zhoujinsong